Skip to content

Introduce Ractor benchmarks and harness #375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 17 commits into from
Oct 28, 2025
Merged

Introduce Ractor benchmarks and harness #375

merged 17 commits into from
Oct 28, 2025

Conversation

@luke-gruber
Copy link
Collaborator

@luke-gruber luke-gruber commented Aug 8, 2025 •
edited
Loading

This PR introduces a set of benchmarks for assessing the performance of Ractors. The main parts of this work are

1. Introduce a set of Ractor specific performance benchmarks

A new harness harness-ractor.rb has been introduced. This harness will run the benchmark code inside a number of Ractors concurrently and collect timing information. By default this is 0 (ractors disabled), 1, 2, 4 and 8.

A number of Ractor specific benchmarks designed to test specific scenarios of multi-ractor mode have been introduced into the benchmarks-ractor directory. They've also been included in benchmarks.yml with the name prefix ractor/.

In order to run these benchmarks with the new harness a new category ractor-only has been introduced. The ractor only benchmarks can be run as follows:

./run_benchmarks.rb --category=ractor-only

2. Adapt existing benchmarks to be Ractor compatible

Existing benchmarks in the benchmarks directory have been updated (where it's possible to do so in a simple manner) to be multi-Ractor safe (mostly making sure any common state is made shareable using Ractor.make_shareable). Benchmarks that have been updated in this way have been marked inside benchmarks.yml with ractor: true.

These tests can be run with either the normal benchmark harness or the Ractor harness, they should be expected to work with both.

A new category ractor has been introduced that will run all of these tests with the Ractor harness alongside the Ractor only tests mentioned above.

All Ractor benchmarks can be run as follows:

./run_benchmarks.rb --category=ractor

Any benchmark can be run with the Ractor harness using the run_once.sh script in conjunction with the HARNESS environment variable

HARNESS=harness-ractor ./run_once.sh benchmarks/activerecord/benchmark.rb

Or a category can be run using the --harness argument to the main benchmark runner

./run_benchmarks.rb --harness=harness-ractor --category=headline

But be aware that running like this will ignore the ractor key in benchmarks.yml and will run all the chosen benchmarks with the ractor harness, regardless of compatibility. Crashes are likely.

3. Harness chain.

For convenience and as an aid to profiling with ractors, this PR also implements the ability to chain harnesses together. For example, running the ractor-only benchmarks through the Vernier profiler can be done as follows:

HARNESS_CHAIN="vernier,ractor" ./run_benchmarks.rb --harness=harness-chain --category=ractor-only

@luke-gruber luke-gruber marked this pull request as draft August 8, 2025 18:31
@luke-gruber luke-gruber force-pushed the ractor-harness-luke branch 2 times, most recently from 9a48fe5 to 607cd5b Compare August 12, 2025 15:39
@eightbitraptor eightbitraptor changed the title [WIP] Ractor harness Introduce Ractor benchmarks and harness Oct 15, 2025
@eightbitraptor eightbitraptor marked this pull request as ready for review October 15, 2025 19:18
@eightbitraptor eightbitraptor self-assigned this Oct 15, 2025
@tenderlove
Copy link
Member

This looks good to me, but I feel I'm missing something. Are the bechmarks just copied fully in to benchmarks-ractor and then made to be Ractor compatible? Also it seems like there were a bunch of images copied, should they have been?

@luke-gruber
Copy link
Collaborator Author

luke-gruber commented Oct 15, 2025 •
edited
Loading

Some benchmark directories were copied into benchmarks-ractor, then made to work with ractors using the ractor harness. Others required just a few small changes so were not copied, but were made to work with the ractor harness. The ractor harness takes care of creating the ractors and running the run_benchmark block in parallel. I believe the images are all from copying optcarrot into benchmarks-ractor. We can remove them if you want.

@tenderlove
Copy link
Member

Some benchmark directories were copied into benchmarks-ractor, then made to work with ractors using the ractor harness. Others required just a few small changes so were not copied, but were made to work with the ractor harness. The ractor harness takes care of creating the ractors and running the run_benchmark block in parallel.

Got it, makes sense.

I believe the images are all from copying optcarrot into benchmarks-ractor. We can remove them if you want.

If they're not buying us anything, I think we should remove them. They look like possible benchmark results? I'd hate for people to be confused by them.

@eregon
Copy link
Member

eregon commented Oct 15, 2025 •
edited
Loading

Regarding OptCarrot, have you seen https://github.com/mame/optcarrot/blob/master/bin/optcarrot-bench-parallel-on-ractor ?

That seems like a nice way to make OptCarrot Ractor-compatible without copying many files.
That specific file could be modified to not run the benchmark (e.g. if $0 != __FILE__) and required by benchmarks/optcarrot/benchmark.rb if using the Ractor harness (like for other benchmarks which were not copied). Or e.g. just copying the .freeze calls in that file.

I think it's best to avoid large copies in the repo, and benchmarks-ractor/optcarrot seems the only offender in this PR.

@k0kubun
Copy link
Member

k0kubun commented Oct 16, 2025

Looks good to me.

Make 30k_methods benchmark Ractor compatible

What was the change? The diff is so large that GitHub doesn't show the diff.

Regarding OptCarrot, have you seen https://github.com/mame/optcarrot/blob/master/bin/optcarrot-bench-parallel-on-ractor ?

If this is possible, I guess we could let benchmarks-ractor/optcarrot/benchmark.rb load optcarrot from ../../benchmarks/optcarrot, use shims like optcarrot-bench-parallel-on-ractor, and run the benchmark.

@eightbitraptor
Copy link
Contributor

Thanks @k0kubun @eregon I hadn't seen the Optcarrot ractor script yet. I'll work on integrating that instead 👍

@eightbitraptor
Copy link
Contributor

Ok. Branch has been updated with revised Optcarrot Ractor benchmark. Thanks all.

eregon
eregon reviewed Oct 19, 2025
View reviewed changes
@eregon
Copy link
Member

eregon commented Oct 19, 2025 •
edited
Loading

I gave it a try and made benchmarks/optcarrot/benchmark.rb Ractor-compatible in #422, which means no copy at all of OptCarrot files. I checked locally and it runs fine on both ruby-master and 3.4.7.
IOW, optcarrot is treated like other benchmarks (e.g. rubykon) which simply use if ENV["YJIT_BENCH_RACTOR_HARNESS"] to make them Ractor-compatible.
If you like it, I think the best would be copy these commits in this PR (e.g. git merge merge --ff-only no-optcarrot-copy), and potentially squash the removal of benchmarks-ractor/optcarrot with the commits adding it to make the history simpler (unless you want to keep that just in case, fine by me).

The only "copied" benchmark is then knucleotide, and that seems fine as-is because it's pretty small and has significant differences with the fork-based version in benchmarks/knucleotide/benchmark.rb.

eregon
eregon reviewed Oct 19, 2025
View reviewed changes
@eregon
Copy link
Member

eregon commented Oct 19, 2025

One more thought is the additions of Ractor.make_shareable even when not using the ractor harness means those benchmarks will fail to run on TruffleRuby (which doesn't define Ractor).
I think a simple solution there is to just define Ractor.make_shareable as noop or freeze in harness/harness-common.rb.
To make it slightly cleaner we could define a top-level make_shareable method so we don't have to define the Ractor constant on Rubies which don't define it.
We can also deal with that after this PR.

@eregon
Copy link
Member

eregon commented Oct 19, 2025

I went through the diff, LGTM in general.

rwstauner
rwstauner reviewed Oct 27, 2025
View reviewed changes
rwstauner
rwstauner reviewed Oct 27, 2025
View reviewed changes
eightbitraptor and others added 2 commits October 27, 2025 16:28
@eightbitraptor @luke-gruber @etiennebarrie
Make most benchmarks Ractor compatible
216c9fa 
Co-Authored-By: Luke Gruber <[email protected]>
Co-Authored-By: Étienne Barrié <[email protected]>
@luke-gruber @eightbitraptor
Make 30k_methods benchmark Ractor compatible
d693594 
Committed seperately so the noisy diff can be easily excluded from
reviews if needed.
eightbitraptor and others added 7 commits October 27, 2025 16:28
@eightbitraptor @luke-gruber
Add Ractor microbenchmarks and harness
4f52349 
These can be run as follows:

ruby -Iharness-ractor/harness.rb benchmarks-ractor/json_parse_float/benchmark.rb

Co-Authored-By: Luke Gruber <[email protected]>
@eightbitraptor @luke-gruber
Update harness API to match harness-ractor
ff4ab27 
Co-Authored-By: Luke Gruber <[email protected]>
@luke-gruber @eightbitraptor
Fix issues with require "bundler/setup"
f6f3c3b 
If your ruby's bundler is different from the saved result in
`Gemfile.lock`, bundler autoswitches by starting a new process. When it
does this it sometimes doesn't pick up the old LOAD_PATH and results in
a LoadError.

Switch to use `Bundler.setup`, which is a simpler process and uses the
current bundler.
@eightbitraptor
Add knucleotide benchmark from the benchmark game
cac1ef2 
2 implementations, one using Process.fork, and one using Ractors - for
comparison
@luke-gruber @eightbitraptor
Add Optcarrot Ractor benchmark
5482d9d 
This is a copy of the Optcarrot benchmark from benchmarks/optcarrot, but
modified to be compatible with multiple Ractors
@luke-gruber @eightbitraptor
Remove the need to specify the full harness path
73e552b 
Instead -Iharness-name can be used
@luke-gruber @eightbitraptor
Implement the ability to use multiple harnesses
5cab2e6 
This commit introduces a harness that can be used to chain two (or more)
harnesses together.

For example. To run a Ractor test with Vernier you can wrap the ractor
harness with the vernier one as follows

HARNESS_CHAIN="vernier,ractor" ruby -Iharness-chain benchmarks-ractor/optcarrot/benchmark.rb
rwstauner
rwstauner reviewed Oct 27, 2025
View reviewed changes
@eightbitraptor @luke-gruber
Introduce ractor and ractor-only categories
b295277 
This commit modifies the benchmark runner to add 2 new categories:

--category=ractor-only Runs only those benchmarks in benchmarks-ractor
using the Ractor harness.

--category=ractor In addition to the ractor-only benchmarks, this will
run benchmarks specified in benchmarks.yml with `ractor: true` using the
Ractor harness.

In addition to this we've modified the burn in and run once scripts to
use the new harnesses and categories too

Co-Authored-By: Luke Gruber <[email protected]>
rwstauner
rwstauner reviewed Oct 27, 2025
View reviewed changes
@eightbitraptor
Run Ractor tests in CI
f057b26 
Also make sure that the benchmarks list test is up to date
@eightbitraptor
Update documentation for new Ractor benchmarks
e191c14
@eregon @eightbitraptor
Unify Optcarrot benchmarks again.
b5591c3 
This commit removes the need to copy any Optcarrot files, by using an
approach based on the optcarrot-bench-parallel-on-ractor script included
in Optcarrot, to make all necessary state shareable when the benchmark
is running under the Ractor harness.

Co-Authored-By: Matt Valentine-House <[email protected]>
@eightbitraptor
Copy link
Contributor

@eregon Thanks for #422 I reallly like the idea of unifying the optcarrot benchmarks in this way. I've removed my previous attempt and pulled in a commit that implements your changes on top of the current work. 👍

@eregon
Copy link
Member

eregon commented Oct 28, 2025

Great, I saw the previous state, it removed the copy of the files so that seemed fine too to me but indeed, I think just ractor: true in YAML and using the same benchmark.rb and Ractor.make_shareable is even cleaner :)

@eregon eregon mentioned this pull request Oct 28, 2025
Closed
@eightbitraptor
Use an Array of floats for the json float benchmark
83f1331 
JSON dictionary keys are always strings, so this benchmark could be seen
to be misleading, as only the values are floats, the keys will be
implicitly converted when the hash is converted to json.

I've chosen to do use an array here instead of explicilty converting the
keys into strings because it avoids the complexity of string
interpolation, and it keeps the original intention of this micro
benchmark which was to have all the contents of the data structure
homogenous.
@eightbitraptor eightbitraptor merged commit 2f7bad3 into main Oct 28, 2025
3 checks passed
@eightbitraptor eightbitraptor deleted the ractor-harness-luke branch October 28, 2025 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eightbitraptor eightbitraptor eightbitraptor left review comments

@rwstauner rwstauner rwstauner left review comments

@eregon eregon eregon approved these changes

@k0kubun k0kubun k0kubun approved these changes

@tenderlove tenderlove Awaiting requested review from tenderlove

Assignees

@eightbitraptor eightbitraptor

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@luke-gruber @tenderlove @eregon @k0kubun @eightbitraptor @rwstauner